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ABSTRACT 

Holistic scoring is widely used to assess writing 
proficiency in Engli sh-as-a-Second-Language (ESL) composition • 
Written recall protocols have recently been used to investigate *„ne 
relationship between how much holistic scorers comprehend of a civen 
te*:t and how high they rate the quality of that text. This study 
investigated whether readers approach a holistic evaluation task 
differently when they are focused (i.e., aware they will ^Iso be 
tested for comprehension) from when they are naive (i.e.. unaware) , 
Subjects were 12 college ESL composition teachers experienced in 
holistic assessment of writing proficiency. Each read, rated , and 
recalled one of two texts , then repeated the cycle with the other 
text. One text was superior to the other . Focused raters recalled 
significantly more overall but were unable to recall more of the 
better-written text. It is concluded that recall studies of the 
holistic reading process should use naive subjects to avoid the 
interference of an ensuing recall task. It is also suggested that 
insights into the holistic reading process can support , extend, and, 
to some extent, reconcile various general reading 
process/comprehensior. models. (KSE) 
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Ab s tr ac t 

Holistic scoring is widely used to assess writing 
proficiency in both English and ESL composition. 
Written recall protocols have recently been employed to 
investigate the relationship between how much holistic 
scorers comprehend of a given text and how highly they 
rate the quality of that text. In this s tu(*y , the 
author examined the issue of whether readers approach a 
holistic evaluation task differently when they are 
focused (i.e., aware they will also be tested for 
comprehension) from when they are naive (i.e., 
unaware). Subjects drawn from a pool of ESL 
composition teachers experienced in holistic assessment 
of writing proficiency read, rated, and recalled one of 
two texts. Then, they exchanged papers and repeated 
the read, rate, recall cycle. Analysis of the two sets 
of recall protocols reveals that, in contrast to their 
naive counterparts, focused raters recalled 
significantly more overall but were unable to recall 
more of the be t ter-wr it ten text. The author concludes 
that recall studies of the ho 1 i 9 .c ic reading process 
should be done with naive subjects to avoid the 
interference caused by awareness of an ensuing recall 
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task and suggests that insights into the holistic 
reading process can help support, extend, and to some 
extent reconcile various general reading 
process/comprehension models. 
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Using Written Recall Protocols to Measure 
Aspects of Holistic Assessment of Second 
Language Wr it ing Proficiency: 
The Effect of Task Awareness 
Holistic scoring, which can be defined in generic 
terms as direct assessment of the overall quality of a 
writing sample (Davis, Scr iven , & Thomas , 1981; White, 
1985), is widely used in American universities to 
evaluate the writing proficiency of both native (NS) 
and non-native (NNS) speakers of English. Although 
the. e are several variations in the procedure (see 
Perkins, 1983; White, 1985), researchers in both 
English and ESL compos it ion general ly agree that 
holistic scoring is both a reliable (Perkins, 1983; 
White, 1985) and valid (Perkins, 1983 ) method for 
providing a direct assessment of writing proficiency. 
However, little has been done to investigate the 
holistic read ing process, itself. Thus, questions 
concerning how holistic raters process test and how 
much they comprehend of what they have read persist 
among both researchers and practitioners in the fields 
of ESL and English composition. 
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One means of investigating the reading processes 
of holistic raters of writing proficiency is the 
written recall protocol. Recall protocols are a 
power f ul tool in the study of prose comprehension. 
Their range of application includes analysis of text 
structure f the study of the organization of information 
in memory , and the study cf how a text is processed 
from one task situation to another (Voes et a 1 . , 
1982) . 

The widespread use of written recall protocols to 
measure reading comprehension supports Connor 1 s (1984) 
assertion that recall is the operational definition of 
comprehension. Indeed, several researchers, including 
Meyer (1973a, 1973b, 1974, 1975, 1980a, 1980b, 1981a, 
1981b, 1982), Meyer and KcConkie (1973 , 1974), 
Christopher son et si. (1981), Carrell (1983 , 1984a, 
1984b, 1986), Bernhardt (1983, 198*, 1985, 1986) and 
Lee (1986a, 19 86b) have relied heavily on recall 
protocols to measure aspects of reading comprehension 
as they relate to the processing of texts by low- 
proficiency native speakers, ESL students at the 
university level, and foreign language learn ers at both 
the secondary and university level. 
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However f a potential drawback in using written 
recall protocols to explore the reading processes of 
holistic raters of second language (L2) writing 
proficiency lies in the question of whether the same 
group of subjects can be used in more than one recall 
protocol study, owing to the possibility that task 
awareness might influence (alter) the way in which they 
process NNS texts in an experimental environment. In 
other words, questions still persist as to whether 
holistic raters might somehow read "differently" or 
more "carefully 11 if they knew they were to be tested 
specifically on the content of what they had read* 
Thus, the issue of whether readers approach a holistic 
evaluation task differently when they are aware they 
will be tested for comprehension from when they are not 
forewarned is one that must be addressed in any 
research into the reading processes of holistic raters 
that may cause subjects to alter their normal reading 
strategies. Moreover, questions initially posed in the 
narrow range of holistic rating of writing proficiency 
may ultimately yield dividends along a much broader 
spectrum of reading research. Specifically, recall- 
protocol research into the reading processes of 
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ho lis tic raters may provide insight & into such aspects 
of the reading process, itself* as tusk awareness, 
depth of processing, and the or&ani nat ion of coitent in 
memory* 

LITERATURE REVIEW 

It is interesting to note that the overwhelming 
majority of research studies concerning test processing 
and the representation of meaning in memory have used 
subjects who vert "focused" (i.e. , who redd a passage 
for the purpose of remembering it) rather than "naive" 
(i.e. , who were unaware they were to be tested on the 
content of the passage). That is to say, regardless of 
whether the task involved writing a summary (e.g., 
Kintsch et al., 1977) or a recall protocol (e.g., 
Meyer, 1982), answering a set of multiple choice 
questions (e.g., Baker & Anderson, 1982), or 
holistically evaluating a composition and then 
rewriting it to more clearly reflect the author 1 s 
intended meaning (e.g., Kaczmarek, 1980), the subjects 
were all aware that they would be tested in one way or 
another on what they had read. 

That subjects in studies of this sort are focused 
is not surprising. Indeed, to attempt to execute a 



9 

ERLC 



8 



Task Awareness 

8 

reading experiment in which the subjects involved .re 
not sensitized to the test environment would likely not 
be feasible owing to the extreme difficulty in 
controlling an experimental setting in which reactive 
factors might cause subjects to approach a reading task 
differently than usual, even if instructed to "read 
normal ly « " 

However, some studies touching upon the difference 
in reading performance of naive readers and of focused 
readers have shed some ligh; on the problem of task 
awareness on reader response. These studies include 
those of Baker and Anderson (1982), Kaczmarek (1980), 

and Lee (1986a). 

In a study designed to investigate the effects of 
inconsistent information on text processing, Baker and 
Anderson (1982) examined whether readers engaged in a 
focused reading task monitor their own comprehension in 
order to determine if ideas expressed in a text are 
consistent. They concluded that while subjects did 
engage in comprehension monitoring when reading 
logically inconsistent texts, they could not 
distinguish well between main ideas and details. The 
results obtained by Baker and Anderson also indicated 
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that specific instructions to read passages carefully 
in preparation for subsequent questions did not seem to 
influence how the subjects approached the reading task. 

In their study, subjects were a war e that they 
would be tested on what they had read. Moreover » some 
subjects were further focused in that they were 
instructed to be alert for inconsistencies in the texts 
they were to read. Nevertheless, Baker and Anderson 
reported that their highly-focused subjects spent no 
more time reading the target passages and were no more 
likely to notice textual inconsistencies during the 
reading process. Baker and Anderson were at somewhat 
of a loss to explain why this was so, although they did 
conjecture that 11 one possible reason for this lack of 
an effect of instructions is that the demand 
characteristics of the task were such that all subjects 
processed the test carefully in preparation for the 
test questions" (p,42) . 

Kaczmar ek { 1 980 ) ind irec t ly addressed the issue of 
focus in a study that com pared teacher assessments of 
writing proficiency with those made by trained raters, 
Kaczmarek's study was designed to assess the validity 
and reliability of two different types of evaluation 
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methods > one subjective and the other objective. In 
hsr study, Kaczmarek used trained raters and writing 
instructors to evaluate essays written by international 
students in an American university. 

la that phase of the study concerning holistic 
evaluation of student responses to a traditional essay 
question, writing instructors were asked to score 
essays on a sis-point proficiency scale* Trained 
raters were also asked to read the essays, but their 
rating task was different. First, they were instructed 
to read each composition for an understanding of the 
writer's intended meaning. Then, they were asked to 
provide a holistic evaluation of the same essays, using 
a six-point scale that focused on comprehensibil ity 
rather than proficiency. 

Kac2mar ek was not looking specifically for the 
differences in recall between naive and focused raters 
of compositions written by KNS college students. 
Nevertheless, she was sufficiently struck by the 
results she obtained to acknowledge that the holistic 
scores of focused raters may veil have been influenced 
by the fact that they were given a comprehension task 
in addition to a rating task. 
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Whether or not their scores were influenced by 
their being focused on the purpose of the reading task, 
however, is by no means clear, as KacztD rek made no 
provisions in her experimental design for investigation 
of that specific quest ion • Nor were her focused readers 
using the same criteria for holistic evaluation as her 
naive readers . In addition, as hers was a study of 
concurrent validity, her focused readers were trained 
holistic raters, but not experienced ESL composition 
teachers like her naive readers. Finally, Kaczmar ek 9 s 
procedures did not include having raters, either naive 
or focused, write recall protocols. Consequently, any 
conclusions drawn would be tenuous at best, especially 
insofar as rater comprehension, per se, was not an 
issue in her study. 

In a few studies, readers were not oriented to 
read for the purpose of recalling the target passage 
(e.g., Carrell , 1 983 , 1984b) . However , it is still not 
clear if holistic raters will, as a matter of course, 
read in the same way when reading specifically to form 
a general impression as subjects read when knowingly 
participating in a recall protocol study. 
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One study that did attempt to explore the question 
of hov task awareness influences amount of recall was 
that of Lee (1986a). Lee used 320 KS American students 
enrolled in Spanish classes in two different 
universities. Consequently* variations in 
instructional procedures and student placement 
presented a potent ially confound ing source of 
variability. The same textbooks, however, were used in 
both programs at the four levels being tested. 

All testing was carried out in the subjects 1 
regular classrooms during regularly-scheduled times , 
although the experimenter was present during testing in 
only four of the 16 classes tested. All instructions 
were given in English, and the instructions contained 
in the test packets were varied so that an equal number 
of subjects were exposed to the respective test 
cond it ions » 

All subjects were instructed to read a passage 
written in Spanish and then write a recall protocol of 
what they had read. Half of the subjects were required 
to produce protocols in Spanish, while the orher half 
wrote in English. Half of each group were given 
prereading instructions alerting them to the fact that 
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they would be required to produce a protocol, while the 
other half received no prereading instructions beyond 
being told to read the passage at their own rat e * 

Protocols were analysed on the basis of the type 
of idea unit matrix recommended by Bransford and 
Johnson (1972)* Recall scores were analyzed by means 
of a three-factor analysis of variance. On the basis 
of this analysis, Lee concluded that "prereading 
directions alone are not sufficient to enhance recall 
of a passage" (p. 208). 

It is tempting to generalize the results obtained 
by Lee to a setting in which trained and experienced 
raters of ESL writing proficiency are engaged in a 
holistic evaluation task. However, it must be 
remembered that Lee's subjects were university students 
performing under ex per imental condit ions in which 
"subjects assume they will be performing some sort of 
task with the content of the passage" (p. 206). 

Subjects' assumptions that the reading task 
involved attention to the content of the passage may or 
may not have been a confounding variable in Lee's 
experiment. However, in a task involving the 
assessment of writing proficiency, attention to content 
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becomes crucial lo the issue of the role of 

compr ehent ion in the process of holistic evaluation. 

Specifically with regard to holistic evaluation, 
whether pr ior knowledge of comprehens ion test ing 
significantly influences how closely one it tends to 
detail in the target passage is a question thar remains 
unanswered. In other words, will a subject read a 
passage differently (i.e., ©ore carefully) f o>* 
comprehension than he/she reads for the purpose of 
holistic evaluation? And will a subject, given such 
prior knowledge, still be able to attend to a holistic 
evaluation task while reading a text for comprehension? 

THE STUDY 

The study reported here is the second of a two- 
part experiment in which subjects hoi is t ically rated 
the quality of texts generated by NKS college students 
and then wrote immediate recall protocols to test their 
compr ebens ion of those texts (See Janopou lo s , 1987 , for 
retails). The first part of the experiment revealed 
that holistic raters of L2 wr it ing proficiency recalled 
significantly more of a better written text than of a 
text judged to be qualitatively inferior (see 

15 



Task Awareness 
15 

Janopoulos, 1989 frr details). The purpose of this, 
the second part of the experiment, was to determine if 
task awareness - specifically, prior knowledge that a 
holistic rating task was to he followed by a test of 
reading comprehension - affects the way in which 
holistic raters of L2 writing proficiency approach 
their reading task* 

In this study, after the subjects had rated and 
recalled a NNS text as naive readers, they were 
instructed to rate and recall a second NNS text* As 
the subjects bad just completed one "read /rate/ recall" 
cycle, they were aware that they would be tested on 
their recall* Thus, in the experiment described here, 
the subjects were focused. Proceeding from the 
assumption that focused holistic raters will approach a 
reading task differently than their naive counterparts, 
this study posed the following two research questions: 

1. Will focused holistic raters recall 
significantly more overall of two NNS texts than their 
naive counterparts, and; 

2. Will focused holistic raters recall 
significantly more of a better written NNS text than of 
a less-well written NNS text? 
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Sub iects 

Twelve ESL composition instructors at the Ohio 
State University participated* Each of the 12 
instruct or s had ex tensive prior exper ience in using 
holistic evaluation procedures to assess L2 writing 
prof ic iency • 

Placement essays written by incoming NNS graduate 
students at Ohio State were used for all phases of the 
study. These essays, assigned as a one hour in-class 
writing task , were rated on a four-point scale that 
corresponds to the three proficiency levels used by 
Ohio State University's ESL Composition course sequence 
(106, 107, 108), plus Q (Qualified to bypass the ESL 
Compos it ion course sequence). 
Proc edur eg 

The Recall Task 
In order to familiarise themselves with the rating 
standards, subjects were provided a list of holistic 
criteria and several model compos it ions one week pr ior 
to the experimental session. The session c ommenced 
with a discussion of the scoring criteria, followed by 
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8 practice rating session during which results were 
discussed and subjects 1 scores were compared • 

Once the researcher was satisfied that subjects 
were using the same general rating criteria, they were 
given a series of ten randomly selected compositions 
one at a time* After reading each text, subjects 
recorded their scores on individual tally sheets. 

Two functions were served in this phase of the 
study, First, in t er rat er reliability (Pearson r) was 
computed by comparing the scores assigned immediately 
after the original writing task with the scores 
assigned by each subject. Interrater reliability 
ranged from a low of .6877 to a high of .9129, with ten 
of twelve subjects achieving a Pearson r correlation 
coefficient in excess of .77. 

Second, the consecutive scoring of several 
compos it ions created an environment which closely 
approximated that of a typical holistic rating session. 
In this way, subjects were conditioned to approach the 
rating task as naive readers, without forewarning of 
the subsequent recall task. 

Once the appropriate holistic scoring environment 
was created, the next phase of the experiment began. 

18 
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Sub J ect s were given an eleventh essay to rate, but 
unlike the previous phase, they were not given the same 
essay. Rather, ha If received a level 107 ( lower 
quality) text and half received a level 108 (higher 
quality) text (see Appendix A). As with all sample 
texts used in the experiment, prejudgments of these two 
texts were based upon initial raters 1 assessments and 
verified independently by the researcher and the 
director of the ESL Composition program. 

Upon completing the holistic scoring task, 
subjects were directed to write a recall protocol of 
what they had read. Specific instruct ions foil owed 
guidelines established by Johnson (1970), with 
paraphrasing allowed when necessary ( Fr eder iksen , 1977; 
Handler, 1970). Subject s were permitted an unlimited 
time to write their protocols, but were not allowed to 
refer to the essays they had just read. 

Next, subjects traded papers and repeated the 
process of reading and rating the sample texts (see 
Figure 1). Upon comp let ion of this task, subjects were 
again instructed to write recall protocols of the 
compositions they had just re?t' . Thus, subjects 
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aoproached the reading/rating task in a focused, or 
forewarned manner . 

The Scoring Task 
Generation of "Negotiated" Texts 

Because the recall task in this study used texts 
written by NNS authors, steps had to be taken to insure 
that each instrument for scoring recall protocols 
accurately reflected the NNS author 1 s intended meaning . 
Accordingly , three independent readers rewrote the 
texts t j reflect what they felt was an accurate 
assessment of each NNS author 1 s intended meaning. 
Rewritten texts were then compared in order to resolve 
differences in interpretation anu reach consensus. 
These "negotiated" texts served as the basis upon *; h i c b 
a scoring matrix for each of the two NNS compositions 
was produced (see Appendix B ) • 
Generation of Weighted Propositions! Counts 

The two "negotiated" t e^t were analysed using 
Meyer's (1973) proposition count procedure. The 
procedure , which acsigns each idea unit (proposition) a 
value of relative importance to the text, is designed 
to extract a recall score f r on a written protocol. 



20 



Task Awareness 
20 

Foil owing this procedure, the researcher first comp iled 
a list of all propositions contained in each text. 
Then, a colleague with experience in the procedure 
compared the lists to the texts and suggested 
modifications. Next, three other colleagues, us ing 
copies of the "negotiated" texts as guides, assigned 
*ach proposition a rating of from 1 (of negligible 
importance to le texJ to 7 (extremely important to 
the text)* The three ratings for each proposition were 
then averaged, wi'«h the resulting list of weighted 
propositional values creating a Protocol Scoring Matrix 
( PSM) , or mean rating scale, for each text (see 
Append ix C ) . 

Scoring of Written Recall Protocols 

Subjects 1 written recall protocols were matched 
with the appropriate idea units to determine the number 
of correctly recalled propositions* The raw scores of 
all the idea units in each protocol were then added to 
compute the raw recall score for that particular 
protocol ♦ 

Scoring reliability for raw recall scores was then 
established by a second rater, who had been extensively 
trained in the procedure. Interrater reliability 
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(Pearson r) between researcher and independent rater 
was computed at .9695. 
Data Analysis 

Raw recall scores were converted into weighted 
recall scores by tallying the values assigned by the 
PSM to each correctly recalled proposition. These 
values were , in turn, converted into percentages by 
dividing them into the sum total of weighted idea units 
in each NNS-au thor ed text. Thus, protocol scores based 
on the higher and lover quality NNS compositions were 
made amenable to comparison. 

A two-way analysis of variance (AN OVA) was then 
computed to compare the mean recall scores of both 
groups of subjects (Focused and Naive) for both NNS 
texts (Lower and Higher ) . Fol lowup procedures employed 
a one-way AN07A • 
Resume 

The null hypothesis for the first research 
question posed by this study predicted that there would 
be no significant difference between the total amount 
of content recalled by focused readers of two NNS texts 
and the total amount of content recalled by naive 
readers of the same texts . 
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Table 1 shove the means and standard dev iat ions 
for holistic reader orientation (Focused versus Naive) 
and level of assessed proficiency (Lower versus Higher) 
of the two NNS texts read and recalled in this study. 
Examination of Table 1 reveals that focused readers 
produced a mean recall score of 52 .07 ( S . D . «1 3 . 82 ) > 
while Naive readers produced a mean recall score of 
35.72 (S.D.«15.16) . The two-way Anova presented in 
Table 2 yielded an F-ratio that was statistically 
significant ( f ( 1 , 20 ) *8 . 56 > p<.01 ) . On the basis of 
these results > the null hypothesis must be rejected. 
Thus, it can be concluded that Focused holistic readers 
as a group recalled significantly more of the two NNS 
texts than their Naive counterparts. 

The null hypothesis for the second research 
question addressed in this study predicted that there 
would be no significant difference between the amount 
of content recalled by Focused readers of the lower- 
rated NNS text and the higher-rated NNS text. Table 1 
shows that Focused readers of the Lower text produced a 
mean recall score of 53.63 ( S #D. »14 .07 ) , while Focused 
readers of the Higher text produced a slightly lower 
mean recall score of 50.50 ( S . D. «14 . 72 ) . A one-way 
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ANOVA (see Table 3) yielded an F-ratio that was not 
statistically significant* On the basis of these 
results, the null hypothesis must be retained. In 
other words, Focused readers recalled both the higher- 
rated and lower-rated texts equally veil. 
Conclusions 

Previous research ( Janopou los , 1987, 1989) has 
revealed that naive holistic raters of NNS wr it ing 
proficiency recall significantly more of a better 
written text than of a text of lower quality. Thus, 
there appears to be a relationship between the quality 
of a NNS text and the amount of content a holistic 
rater of writing proficiency recalls of that text. 

The focus of this study was on the influence of 
task awareness on readers engaged in a holistic scoring 
procedure involving NNS texts. The study posed two 
questions : 

1. Would focused readers recall significantly 
more overall than naive readers? and : 

2. Would focused readers be able to recall 
significantly more of a better vr i t ten NNS text 
than of a NNS text of inferior quality? 
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Analysis of the data produced in this experiment 
indicate that task awareness alters the way in which a 
reader approaches a holistic scoring task. 
Specifically, results indicate that focused subjects 
recalled more of the texts they assessed for writing 
proficiency than did their naive count erpart s , but they 
were unable to recall significantly more of the better 
written text* In other words, recall was enhanced, but 
the ability to make a qualitative (i.e., content 
recalled) distinction between NNS texts of differing 
qualities was lost. 
Discussion 

In a narrow sense, the intent of this study was to 
determine if focused holistic raters of NNS writing 
proficiency performed in a comparable manner to their 
naive counterparts with respect to (1) their overall 
comprehension of a pair of NNS texts, and (2) their 
ability to recall more of a better written NNS text » 
On the basis of the results reported above, it must be 
concluded that in studies of the reading processes of 
holistic raters of writing proficiency that use written 
recall protocols to measure reading comprehension, 
subjects should be naive; that is, unaware that they 
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will be tested on their recall of the content of the 
texts they have to read. 

In a broader reuse » however , the results of this 
study raise basic questions cor cerning the reading 
process, itself. Now that evidence has been provided 
that task awareness affects the way in which a holistic 
reader approaches a text, we need to inves t igat e why 
that is so, what strategies are employed, and how those 
strategies work. 

A wide variety of reading process/comprehension 
models have been advanced by researchers in the fields 
of cognitive psychology, information processing, and 
psychol inguist ics (e.g., LaBerge & Samuels, 1974; 
Goodman, 1 97 6 i Kintsch & van Dijk, 1978; Stanovich , 
1980; Rumelhart , 1980; Miller & Kintsch, 1980; Just & 
Carpenter , 1980 ; Bernhardt , 1986). In many ways , these 
mode 1 s offer conflicting accounts of the read ing 
process , but in some areas they are by no means 
c ontrad ic tory . This is especially true in the area of 
comprehension monitoring, where , as Baker and Brown 
(1984) note p "comprehension monitoring activities are 
implicitly, if not explicitly, incorporated into 
several recent models of comprehension" (p .355) . 
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Moreover, insights into the mechanisms of 
comprehension monitoring provided by research in one 
field can complement and extend the descriptive power 
of a reading process/ comprehension model in another * 
An example ;f this can be seen in the applications of 
the construct of me tacognit ion discussed by Baker, 
(1979) and Brown (1980) to the read ing mo del of LaBerge 
and Samuels ( 197 4) ♦ 

Samue Is and Kami 1 (1984) observe that the LaBerge 
and Samuels model, in both its early and revised forms, 
speaks of how a reader's limited attention capacity can 
by severely taxed by "the combined demands of decoding 
and comprehension," thereby resulting in what fchey term 
"attention switching" (p • 197 ) • Once the attention 
switching mechanism is activated, a shift in the 
allocation of processing opacity is initiated, and the 
skilled reader presumably applies any or all of what 
Brown (1980) identifies as the three main types of 
me tacognit ive skills - awareness, monitoring, and 
deployment of compensatory skills - to the reading 
task. Furthermore, it is plausible that attention 
switching can be triggered by other task demands than 
decoding and comprehend ing . For example , the attention 



ERLC 



27 



Task Awareness 
27 

switching phenomenon posited by LaBerge and Samuels 
might also be seen as the trigger that activates the 
mechanism by which skilled readers employ different 
metacognit ive strategies when reading for meaning 
(i.e., comprehension monitoring) than when reading for 
remembering (e.g., identifying important ideas and 
testing one's mastery of material). 

In the present study, the fact that focused raters 
recalled significantly more overall but were unable to 
recall more of the better written of the two NNS texts 
may be explained in the following way: the multiple 
task demands imposed upon focused raters resulted in 
attention switching, which not only affected their 
allocation of pro ces sing capacity, but also influenced 
the selection of those me t acogn i t ive strategies and 
skills appropriate to both comprehension and recall. 

The naive holistic raters in this study clearly 
read with comprehension, perhaps because when 
holistically assessing writing proficiency, readers 
cannot accomplish the latter without doing the former. 
To accomplish this , they likely employed the 
metacognit ive strategy of comprehension monitoring. 
Focused raters, on the other hand, may well have 
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mobilised the metacognitive strategies associated with 
reading for remembering in addition to comprehension 
monitoring, which, in turn, resulted in better overall 
comprehension of both the lower and higher quality 
texts . 

Yet their enl a need recall was apparently achieved 
at the expense of the ability to make significant 
qualitative distinctions between the two NNS texts, 
perhaps because a lesser amount of meaning needs to be 
accessed in order to activate the metacognit ive skills 
needed to read for remembering. Moreover, perhaps in 
reading for remembering, comprehension becomes a means 
rather than an end in itself* 

CONCLUSION 

Clearly, the results reported in this study raise 
more quest ions than they answer, especially with regard 
to the reading strategies employed by holistic scorers 
of writing proficiency. Moreover, this study focused 
on an extremely specialized holistic reading task - 
native English speakers reading texts written by con- 
native Engl ish speakers - and considered only one 
variable in that task - prior knowledge of an ensuing 
test of recall. Further research that investigates the 
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holistic reading process, itself, more directly (using, 
for example » eye- tracking technology or comp let e 
propo sit ions 1 ana lyses of recall protocols) is also 
needed to provide a clearer picture of the influence of 
task demands on not only holistic reading for tbe 
purpose of assessing writing proficiency, but all other 
types of reading activities as veil. 
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Table 1 

Neons and Standard Deviations of Recall Scores by Reader 
Orientation and Prol iciency Assessment 



Reader 
Orientation 



Assessed Proficiency 



Naive 



Focused 



Overall 



Lower 
n x a 


Higher 
n x cr 


Overall 
n x cr 


6 27.28 9.09 


6 44.17 15.86 


12 35.72 15.16 


6 53.63 14.07 


6 50.50 14.72 


12 52.07 13.82 


12 40.45 17.80 


12 47.34 14.96 


24 43.89 16.46 
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Table 2- 



Analysis of Variance of Reading Comprehension by Reader 
Orientation and Proficiency Assessment 



Source 


df 


SS 


XS 


F 




Prof i c i enc v 

Assessment 

(A) 


1 


284. 1440 


284.1440 


1 .52 


0.2324 


Reader 

Orientation 

(B) 


1 


1602.9541 


1602.9541 


8.56 


0.0084* 


Proficiency 
by 

Orientation 
(AB) 


1 


601.6010 


601.6010 


3.21 


0.0883 


Error 

f S/AB) 


20 


3746.8225 


187.3411 






Total 


23 


6235.5218 









*p < .01 
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Table 3 

Analysis of Variance of Reading Co mprehension 
"Focused /Higher bv Focused/Lower) 

Source df SS MS £ £ 

(aJb X 2 /A 2 B 2 ) 1 29.4220 29.422C 0.14 .7143 



Error 


10 


2074.3450 207.4345 


Total 


11 


2103.7670 
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Sub -1 ects Reader Orientation* 

Focused 
H 
L 
H 
L 
H 
L 
H 
L 
H 
L 
H 
L 

* 1 ■ Lower-rated (107) text 

H = Higher-rated (108) text 
Figure 1. 

O rder of administration of Naive and Focused reading 
tasks with respect to target texts 
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Appendix A 
Original Texts (Levels 107 & 108 ) 

Conposition *1, Level 107 

For giving an answer to the question if it's better 
to learn to be competitive or cooperative it is nessessary 
to consider a few faktors which are important 

- In which society do the people live and how it is 
organized 

- the personelity of the individual and the individual 
abil it ies . 

Evervone will agree with the statement that it is 
important to learn both of these abilities, to be 
competitive and cooperative. 

And everyone will agree that every society has 
developed its own way to selekt people at school in 
business everywhere . 

Ther are differences between the social aims with an 
influence on the individual aims. In America the 
organization of the society is much more competitive 
oriented as in some states of the eastern hemisphere. 
And as far as I know are the people in Japan much more 
oreintated in a cooperative style of working and living 
but they are very competitive as a nation or as a 
company - as every kind of a group. 
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The reasons for these different kinds of social 
orgainzation can be found in the historical development 
and the conditions of living. 

Japan, for many years isolated from any influence and 
with little space for too many people was forced to 
develop a very cooperative style of living. The 
development of the USA was determind of individual 
activities. So they developed a more competitive style 
of social orgainization . 

I t is important to see these historical and social 
cri terias before answering the question if its better to 
learn a competitive or cooperative style of living . No 
individual person can be seen isolated from the society to 
which it belongs . 
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Corr.posi tion -2, Level 108 

1 think this is a rather controversial problem. It 
is difficult for me to say that one is definitely better 
than the other since it may depend on the importance of 
the matter of the situation. 

There are times when taking a quick and decisive 
action is more desirable. For instance, you got a very 
serious disease and your doctor told you that you need an 
operation which is very dangerous. You were also told 
that you might die if you don't have the operation. In 
such a case, it would be very hard to make a quick 
decision, but you should do so since the matter is very 
urgent . For another example , I go to a restaurant with 
some of my friends and we are supposed to have only 30 
minutes in that restaurant because we are going to a movie 
afterwards. In that particular situation, I have to make 
a quick decision for my own sake . Otherwise, I may not be 
able to finish my meal or miss some part of the movie . 

There are many other cases when it is much better to 
think something over carefully , rather than to make a 
hasty choice . It is more advi sable to be careful 
especially when you have to make a very important deci sion 
in your lif e . For example , when you decide what you ' 11 
study in college , you should take time and think over and 
over since your major will be related to your career. 
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For another example, 1 can say when you choose your 
spouse. That kind of decision is what you have to live 
with it all your life and what you can not change easily 
even though you want to. 

In conclusion, I would say that being carefull and 
taking enough time is more preferrable if the issue is not 
very urgent and doesn't need to be done right away because 
you'll probably make less mistakes and repent less if you 
do so. 
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Appendix B 
Negotiated Texts 
COMPOSITION M ( LEVEL 107 

In order to give an answer to the question of whether 
it's better to learn to be competitive or cooperative it 
is necessary to consider a few factors which are important: 

- In which society do the people live and how it is 
organized > 

- The personality of the individual and the 
individual abilities. 

Everyone will agree with the statement that it is 
important to learn both of these abilities , to be 
competitive and cooperative . 

And everyone will agree that every society has 
developed its own way to select people at school and in 
business everywhere . 

There are differences between the social aims that 
have an influence on the individual aims. In America the 
organization of the society is much more competitively 
oriented than in some states in the eastern hemisphere . 
As far as a know the people in Japan are much more oriented 
in a cooperative style of working and living but they are 
very competitive as a nation or as a company - whenever 
they function as a group . 
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The reasons for these different kinds of social 
organization can be found in the historical development 
and the conditions of living. 

Japan, isolated for many years from any influence and 
with little space for too many people, was forced to 
develop a very cooperative style of living. In contrast, 
the development of the USA was determined by individual 
activities. So Americans developed a more competitive 
style of social organization. 

It is important to see these historical and social 
criteria before answering the question of whether it 1 s 
better to learn a competitive or cooperative style of 
living. No individual person can be seen as isolated from 
the society to which he/she belongs . 



ERIC 



48 



Task Awareness 
48 

COMPOSITION *2 (LEVEL 108 

I think the question of quick action versus 
deliberation is a rather controversial problem. It is 
difficult for me to say one is definitely better than the 
other since the choice between alternatives may depend on 
the importance of the matter of the situation. 

There are times when taking quick and decisive action 
is more desirable. For instance, you got a very serious 
disease and your doctor told you that you need an operation 
which is very dangerous. You were also told that you might 
die if you didn't have the operation. In such a case, it 
would be very hard to nuke a quick decision, but you should 
do so since the matter is very serious. For another 
example, I go to a restaurant with some of my friends and 
we are supposed to have only 30 minutes in that restaurant 
because we are going toa movie afterwards . In that 
particular situation, I have to make a quick decision about 
what to order for my own rake. Otherwise , 1 may not be 
able to finish my meal or may miss some part of the movie. 

There are many other cases when it is much better to 
think something over careful ly , rather than to make a hasty 
choice . It is more advisable to be careful , especially 
when you have to make a very important decision in your 
life. For example , when you decide what you'll study in 
college , you should take time and think it over and over 
since your major will be related to your career . 
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For another example, 1 can say when you choose your 
spouse. That kind of decision is what you have to live 
with all your life and what you can not change even though 
you want to. 

In conclusion, I would say that being careful and 
taking enough time is preferable if the issue is not very 
urgent and action doesn't need to be taken right away 
because you'll probably make fewer mistakes and repent 
less if you do so. 
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Appendix C 



Protocol Scoring Matrices lor Higher ( FORM A) 
and Lower (FORM B) Texts 



FORM A, Text *2 
WEIGHTED SCORES 

1. 3.33 

2. 5 



3. 
4. 

5. 
6. 

7. 
8. 



r 



3.33 



3.67 
6.33 



5.67 



10. 5.33 
U. 4.33 
12. 5.33 



13 
14, 
15. 
16. 



5.33 
4.67 



5.33 



IDEA UNITS 
I think (OPINION) 

the question of quick action 
versus deliberation is a problem. 

The problem is rather controversial 

It is hard to say one is definitely 
better than the other. . . 

Since (Because) 

The choice between alternatives may 
depend on the importance of the 
situation. 

Sometimes taking quick and decisive 
action is more desirable. 

For instance ( EXAMPLE #1) 

You have a disease. 

The disease is very serious. 

Your doctor told you . 



ii it 



17. 3.67 



told you THAT YOU NEED AN 
OPERATION. 

The operation is dangerous. 
(BUT) 

... You may die without it. 

In this case, it would be very hard 
to make a quick decision. 

But 
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18. 4^33 You should do so... 

lS) ' - Since... (Because) 

20. 4_^67 ... t h e matter is very urgent. 

21. 4 . 33 For another example (EXAMPLE #2 ) . 

22 * 1 {the author) go to a restaurant. . . 

23 • ... with some of my friends. . . 

24. ^33 ... and we only have 20 minutes to eat 

25# ^- 6 Z Because. . . 
26. 5 



... we are going to a movie afterwards 

21 ' - In th *s situation, I have to make a 

quicv decision. 

28, ^1 The decision is about what to order. 

29 • lill The decision is FOR MY OWN SAKE. 

Otherwise (if I don't make a quick 
decision) ... 

I may not be able to finish my meal... 
... or 

... I may miss part of the movie. 
(ON THE OTHER HAND) 



30. 4 



31. 3.67 

32. 3 



33. .4 



34. '3.33 



35 ' There are cases when it is much better 

to think something over carefully 

36 • ii-L 33 " " carefully RATHER THAN 



37. 4.67 



38. 5,33 



... making a hasty choice . 
It is more advisable to be careful... 

39 * ^2 " " careful WHEN YOU HAVE TO MAKE A 

VERY IMPORTANT DECISION IN YOUR LIFE. 

40. 4^33 For example (EXAMPLE #1) 
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41 . a 



42. 4.67 

43. 5 



44. 3.67 

45. 4 



46 . 4.33 

47. 3.33 

48. 5,67 

49. 5 



50. 5.67 

51. 5 

52. 4 



53. 3.67 

54. 4.33 

55. 5.67 

56. 6 



57. 4 



58. 5 



59. 4.33 



60. 4.67 

61. 4 



62. 5 



You should Cake your time to think it 
over . . . 



M II 



think it over WHEN YOU DECIDE 



" M decide WHAT YOU'LL STUDY IN 
COLLEGE. . . 

Since (Because) 

Your major will be related to your 
career. 

For another example ( EXAMPLE #2) 

Take your time ... 

... when you choose your spouse. 

You have to live with that kind of 
decision. . . 



ii ii 



decision ALL YOUR LIFE. . . 



... and you cannot change ( that 
decision) . . . 

... even though (if) you want to, 

In conclusion. . . 

• • • being careful 

... and taking enough time 

• • • time IS PREFERABLE 

... if 

... the issue is not very urgent 



5 1 M 



TO Rp^oMr f ND - (a< T tion) DOESN'T NEED 
TO BE DONE ( taken ) 

... right away (immediately) 
• ■ . because 

... you'll probably make fewer 
mistakes 
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" " make fewer mistakes AND REPENT LESS 
... if 

if YOU DO SO (take your time). 

1= TOTAL 
295.65 



63. f* 

64. 3.33 

65. 3.67 
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FORM 5, Text gj 
WEIGHTED SCORES 
1 . 4.67 • 



2. 


4 . 


33 


3. 


3 




4. 


4. 


67 


5. 


4. 


67 


6. 


6 




7. 


6 




O . 


3. 


67 


9. 


6. 


33 


10. 


5. 


67 


11 . 


2 . 


33 


12. 


4. 


33 


13. 


2 




14. 


3. 


_67 


15. 


3. 


33 


16. 


3 




17. 


3 




18. 


2 . 


33 


19. 


5. 
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IDEA UNITS 

It is necessary to consider a few 
factors . 

" " a few factors WHICH ARE IMPORTANT. 

In order to give an answer to the 
question . . . 

" " of whether it is better to learn 
to be competitive... 

... or cooperative. 

In which society do the people live? 

How is it organized? 

(AND) 

The personality of the individual. 

Individual abilities . 

Everyone will agree with the 
statement . . . 

" " the statement THAT IT IS IMPORTANT 
TO LEARN BOTH OF THESE ABILITIES 
(cooperation and competitiveness). 

Everyone will agree... 

" " agree THAT EVERY SOCIETY HAS 
DEVELOPED ITS OWN WAY... 

" " its own way TO SELECT PEOPLE... 

" " to select people AT SCHOOL. . . 

" " IN BUSINESS. . . 

" " EVERYWHERE . 

There are differences between the 
social aims . . . 
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20. 5.33 



21. 5.67 

22. 6 



23. 4.6 7 

24. 1.33 

25. 6 



26. 6.33 



27. 5.33 

28. 4.67 

29. 6.33 

30. 3.33 

31. 5.33 

32. 6 



33. 4.67 



34. 5 



35. 5.67 

36. 5.67 



37. 4 



38. 5.67 



39. 3.67 

40. 4.67 



" " aims WHICH INFLUENCE INDIVIDUAL 
AIMS . 

In America . . . 

organization of society is much more 
competitively oriented... 

" " oriented THAN IN SOME STATES IN 
THE EASTERN HEMISPHERE. 

As far as I know. . . 

In Japan . . . 

" " the people are much more oriented 
in a cooperative style of working... 

it it 



of working AND LIVING, 



but 



they are very competitive as a nation 
or 

as a company - 

whenever they function as a group . 

There are reasons for these different 
kinds of social organizations . 

These reasons can be found in the 
historical development . . . 

" " and in the conditions of living. 

Japan has been isolated from outside 
influence . . . 



ii it 



• i •« 



influence FOR MANY YEARS... 



and has had little space for too 
many people . . . 



it ii 



people FOR MANY YEARS. 
(CONSEQUENTLY) 
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42 . 


4 


.67 


43. 


5 


.67 


44. 


5 




45. 


6 




46 . 


4 , 


, 33 


47. 


4 




48. 


4 




49. 


3. 


67 


50. 


2. 


33 



ZZ^Z=l TOTAL 
229.34 



Japan was forced Co develop a very 
cooperative style of living. 

In contrast. . . 

The development of the USA was 
determined by individual activities. 

So (consequently) 

Americans developed a more competitive 
style of social organization. 

It is important to see these 
historical and social criteria; . . 

" " criteria BEFORE ANSWERING THE 
QUESTION OF WHETHER IT IS BETTER TO 
LEARN TO BE COMPETITIVE OR COOPERATIVE . 

No individual person can >e seen as 
isolated from. . . 

" n from THE SOCIETY. . . 

" " society WHICH HE/SHE BELONGS TO. 



ERIC 



57 



